Method, device and sensor system for monitoring the surroundings for a vehicle

ABSTRACT

A method for monitoring vehicle surroundings, including: reading in first and second surrounding-area information-items respectively with first and second detectors. The first and second surrounding-area information-items represent information items receivable from the vehicle surroundings and is about at least one object in the surroundings; processing the first surrounding-area information-item with the first detector, to provide first sensor-data, and processing the second surrounding-area information-item with the second detector, to provide second sensor-data; merging the first sensor-data and the second sensor-data, using a time-difference information item, to provide a measuring-signal. The time-difference information item represents a time-difference between a first latency-time, which is needed by the first detector for the processing, up to the provision of the first sensor-data, and a second latency-time, which is needed by the second detector for the processing, up to the provision of the second sensor-data. The measuring signal represents monitored surroundings of the vehicle.

RELATED APPLICATION INFORMATION

The present application claims priority to and the benefit of Germanpatent application no. DE 10 2018 216 809.1, which was filed in Germanyon Sep. 28, 2018, the disclosure of which is incorporated herein byreference.

FIELD OF THE INVENTION

The present invention is based on a device or a method according to thedefinition of the species in the independent claims. A computer programalso constitutes the subject matter of the present invention.

BACKGROUND INFORMATION

In vehicles, several sensors may be used for monitoring thesurroundings. In this connection, it may be necessary to merge sensordata of a plurality of sensors, in particular, for object detection.

SUMMARY OF THE INVENTION

Against this background, using the approach introduced here, a method,in addition, a device which utilizes this method, a system and, finally,a corresponding computer program, according to the main claims, are putforward. Advantageous further refinements and improvements of the deviceindicated in the independent claim are rendered possible by the measuresspecified in the dependent claims.

According to specific embodiments, latency times of a plurality ofparticipating sensors may be taken into account, in particular, in thecase of sensor data fusion; the latency times being sensor-specific or,in other words, specific to a respective, participating, physical andelectrical part of the detector. For example, the latency times may beascertained continuously during sensor operation. In this context, thelatency times may include, in particular, signal propagation times andoverall times of signal processing.

According to specific embodiments, in particular, exact real-timemonitoring of surrounding-area data may be rendered possible in anadvantageous manner. In this connection, for example, reliablesynchronization to a system time may be achieved. Even in the case of acombination of different physical, measurement-dependent tolerances andpropagation times, effects of environmental conditions, signalconversion, error control, data interpretation, etc., which may beheterogeneous in general and difficult to measure in particular, datamay be provided, for example, rigorously and/or accurately,deterministically and also synchronizably and/or accurately timed withregard to a reference time. In particular, for time-sensitive safetyapplications, synchronized data may be made available, in order thatthey may also be acquired in a comprehensible and verifiable manner. Inthe case of sensor data fusion between two different measuringprinciples, for example, a lack of sharpness, which is based on theaddition of tolerances, may be reliably prevented. In addition, a lackof sharpness may be prevented from increasing on the basis of time andpossible degrees of freedom. Therefore, for example, subjects, such astimely functioning, fault-tolerant systems, solid performancecharacteristics and systems, which may also ensure requiredfunctionality in the case of a fault, may be addressed in acomprehensive safety plan. In this connection, in particular,information items available through data technology may be renderedcomparable with regard to their actually real, effective occurrence, andan age of an information item in the system may be known within adefined time window or real-time window.

A method for monitoring the surroundings for a vehicle is put forward,the method including the following steps:

reading in a first surrounding-area information item with the aid of afirst detector and a second surrounding-area information item with theaid of a second detector; the first surrounding-area information itemand the second surrounding-area information item representing aninformation item, which is receivable from the surroundings of thevehicle and is about at least one object in the surroundings;

carrying out processing of the first surrounding-area information itemwith the aid of the first detector, in order to provide first sensordata, and carrying out processing of the second surrounding-areainformation item with the aid of the second detector, in order toprovide second sensor data; and

merging the first sensor data and the second sensor data, using atime-difference information item, in order to provide a measuringsignal; the time-difference information item representing a timedifference between a first latency time, which is needed by the firstdetector for the processing, up to the provision of the first sensordata, and a second latency time, which is needed by the second detectorfor the processing, up to the provision of the second sensor data; themeasuring signal representing monitored surroundings of the vehicle.

This method may be implemented, for example, as software or hardware oras a mixture of software and hardware, in, for example, a control unit.In the method, time data from infrastructure, such as time stamps of atracking device, satellite data or other physical effects, which includetime information, may be considered. Such time data may contribute tothe setting-up of a temporally correct inertial system.

A detector may be configured to monitor the surroundings, usingelectromagnetic or acoustic waves reflected by an object. Thus, asurrounding-area information item may be an electromagnetic or acousticsignal. A surrounding-area information item may represent at least onephysically measurable characteristic of the at least one object in thesurroundings. A detector may be, for example, a camera, an ultrasonicsensor or a radar sensor. The two detectors may be identical ordifferent. The detectors may also use different physical variables formonitoring the surroundings. The information item receivable from thesurroundings of the vehicle may be a representation of the object. Inthe step of processing the surrounding-area data, for example, objectdetection, distance measurement or the like may be carried out. Thus,the sensor data may include data regarding the object, which may beused, for example, by a driving assistance system of the vehicle. Themeasuring signal may include merged data from the first and secondsensor data. In this context, using the time-difference informationitem, it may be ensured that temporally matching sensor data are mergedwith each other. A latency time, which is needed by a detector for theprocessing, up to the provision of sensor data, may represent a periodof time between an instant of obtaining a surrounding-area informationitem, up to an instant of providing sensor data; the period of timebeing specific to the detector in question and, additionally oralternatively, being ascertained continuously. In the fusion step,sensor data fusion of the sensor data may be carried out, using a fusionmodel, a maturity model or the like.

According to one specific embodiment, the method may include a step ofascertaining the time-difference information item. In this connection,in the ascertaining step, signal propagation times, times forphysical-to-electrical conversion, and times for signal processing maybe combined to obtain the first latency time with regard to the firstdetector and the second latency time with regard to the second detector.The step of ascertaining may be executed continuously, for example,during a trip of the vehicle or during the execution of the other stepsof the method. An advantage of such a specific embodiment is that anexact and current latency time may be provided for each detector, whichmeans that a reliable time-difference information item is available forthe sensor data fusion, in order to render real-time monitoringpossible.

In the fusion step, the first sensor data and, additionally oralternatively, the second sensor data may also be scaled, using thetime-difference information item. In addition, or as an alternative, inthis connection, the first sensor data and the second sensor data may besynchronized, using the time-difference information item and a model forsensor data fusion. The model for sensor data fusion may represent amaturity model or another model. Such a specific embodiment provides theadvantage that streams of sensor data may be allowed to flow from thedetectors.

In addition, a neural network, a classification method, a stochasticmethod, a Kalman filter, fuzzy logic and, additionally or alternatively,logic operations, may be used in the fusion step. Such a specificembodiment provides the advantage that the sensor data fusion may becarried out in a reliable, rapid and exact manner.

In addition, in the step of carrying out the processing, the firstsensor data may be provided with a first time information item, and thesecond sensor data may be provided with a second time information item.In this connection, in the fusion step, the first sensor data and thesecond sensor data may be merged, using the first time information itemand the second time information item. The time information item mayrepresent a time equivalent; the time equivalent being able to representa time stamp and, additionally or alternatively, a message counter,communication counter, or the like. An advantage of such a specificembodiment is that a time equivalent suitable as a function of theapplication and suited to exact synchronization may be applied to thesensor data.

According to one specific embodiment, the method may include a step ofemitting a first sensing signal with the aid of the first detector and,additionally or alternatively, emitting a second sensing signal with theaid of the second detector, into the surroundings of the vehicle. Inthis connection, the first surrounding-area information item may bereceivable in response to the first sensing signal. In addition, or asan alternative, the second surrounding-area information item may bereceivable from the surroundings of the vehicle in response to thesecond sensing signal. In this connection, the first detector may have,for example, a principle of detection based on lidar or radar.Additionally or alternatively, the second detector may have, forexample, a principle of detection based on lidar or radar. The firstsensing signal and, additionally or alternatively, the second sensingsignal may include electromagnetic waves. Such a specific embodimentprovides the advantage that a representation of the surroundings of thevehicle may be obtained in a rapid and accurate manner. A signalpropagation time may be calculated from the frequency of emitted andreflected light; acquisition with the aid of lidar being very rapid, andthe acquisition of light at a high frequency, with wavelengths in thenanometer range, also being more rapid than in the case of lowerfrequencies, with wavelengths in the micrometer range.

In the step of carrying out the processing, the first sensor data andthe second sensor data may also be provided with an integrityinformation item. In this connection, the integrity information item maybe checked in the fusion step. The integrity information item mayrepresent encryption of the sensor data, a signature attached to thesensor data, or the like. An advantage of such a specific embodiment isthat incorrect or corrupted data may be rejected or, in the case ofhighly available systems, may be denoted data-technology-specifically bysignal qualifiers as having less integrity. This may be useful, inparticular, when unsharp logic, such as fuzzy logic or neural networks,is used, or when many sources are used for checking plausibility, whichmeans that a signal availability may be increased. In addition to theintegrity information item, a time information item may be used, inorder to check a timeliness of the sensor data. A correctness orintegrity of the data source, that is, the system is free of defectsand/or is operated in a nominal range of application, an integrity andtimeliness of the information, may be provided as a quality criterionfor further processing. Alternatively, in a further processing step, forexample, activation of an actuator may be evaluated in a comparablemanner. As an alternative to checking at the input of the processingunit, the information may be used in order to analyze logicalprocessing, before it is switched to the actuator. This has theadvantage that when they are correct, timely, etc., the data are allowedthrough to the actuator, but the nominal data stream is not held up,before the previous data stream, e.g., input data, have been checked.

In addition, in the reading-in step, a further surrounding-areainformation item may be read in with the aid of a further detector. Thefurther surrounding-area information item may represent an informationitem, which is receivable from the surroundings of the vehicle and isabout at least one object in the surroundings. In this context, in theexecution step, processing of the further surrounding-area informationitem may be carried out with the aid of the further detector, in orderto provide further sensor data. In this connection, in the fusion step,the further sensor data may be merged with the first sensor data and,additionally or alternatively, with the second sensor data, using thetime-difference information item. The time-difference information itemmay represent a time difference between the first latency time, thesecond latency time and, additionally or alternatively, a furtherlatency time, which is needed by the further detector for theprocessing, up to the provision of the further sensor data. Such aspecific embodiment provides the advantage, that a plurality ofdetectors may be, or may become involved in the monitoring of thesurrounding area, in order to allow accurate and reliable monitoring ofthe surrounding area.

In addition, in the reading-in step, the first surrounding-areainformation item may be used for data information. Upon receipt of thesecond surrounding-area information item or a further surrounding-areainformation item, in the negative case, the first surrounding-areainformation item may be disqualified from further processing, before thesame is transmitted to an actuator. Consequently, a diagnosticinformation item or correction information item may overtake a nominaldata stream and/or processing of useful payload data.

Thus, the method may include a step of checking a consistency of thefirst surrounding-area information item and the second surrounding-areainformation item. In a routing step, the measuring signal or the firstsurrounding-area information item may be passed on, if there isconsistency between the first surrounding-area information item and thesecond surrounding-area information item. In a blocking step, themeasuring signal or the first surrounding-area information item may beblocked, if there is no consistency between the first surrounding-areainformation item and the second surrounding-area information item.Consistency may be present, if the two surrounding-area informationitems may be assigned to the same type of object. Consistency may not bepresent, if the two surrounding-area information items may be assignedto different types of objects. In this context, it is advantageous thata correctness information item relating to the consistency may becommunicated through the systems more rapidly than the payload dataundergoing processing. Consequently, it is possible to overtake theprocessing of payload data. This may also be combined highly effectivelywith an approach, in which there is two-way monitoring of redundantcontrol units, via which a communication is exchanged. Here, thismechanism has the advantage that the diagnostic information may renderthe two-way monitoring more effective, owing to the more rapidcommunication.

The approach put forward here also provides a device, which isconfigured to perform, control and/or implement, in correspondingdevices, the steps of a variant of a method put forward here. Theobjective of the present invention may be achieved quickly andefficiently by this embodiment variant of the invention in the form of adevice, as well.

To this end, the device may include at least one processing unit forprocessing signals or data, at least one storage unit for storingsignals or data, at least one interface to a sensor or an actuator forreading in sensor signals from the sensor or for outputting data signalsor control signals to the actuator, and/or at least one communicationsinterface for reading in or outputting data, which are embedded in acommunications protocol. The processing unit may be, for example, asignal processor, a microcontroller or the like; the storage unit beingable to be a RAM, DRAM, etc., or a flash memory, an EEPROM or a magneticstorage unit. The communications interface may be configured to read inor output data wirelessly and/or in a line-conducted manner; acommunications interface, which is able to read in or output theline-conducted data, being able to read in these data, e.g.,electrically or optically, from a corresponding data transmission lineor to output them to a corresponding data transmission line.

In the case at hand, a device may be understood to be an electricaldevice, which processes sensor signals and outputs control and/or datasignals as a function of them. The device may include an interface,which may take the form of hardware and/or software. In a hardwareconfiguration, the interfaces may, for example, be part of a so-calledsystem ASIC, which includes various functions of the device. However, itis also possible for the interfaces to be separate, integrated circuitsor to be at least partially made up of discrete components. In asoftware configuration, the interfaces may be software modules that arepresent, for example, in a microcontroller, next to other softwaremodules.

In one advantageous refinement, the device monitors the surroundings fora vehicle and provides a measuring signal, which represents themonitored surroundings. The device may be configured to provide themeasuring signal to at least one assistance system of the vehicle. Tothis end, the device may access, for example, sensor signals, such assensor data of the detectors, and signals of other vehicle sensors, suchas a speed sensor, a position sensor, a temperature sensor and the like.In the case of use by at least one assistance system of the vehicle, themeasuring signal may be suitable for effecting activation,parameterization, or the like of the assistance system.

A sensor system for monitoring the surroundings for a vehicle is alsoput forward, the sensor system including the following features:

a first detector;

a second detector; and

a specific embodiment of the device mentioned above; the device beingconnected to the first detector and to the second detector so as to beable to transmit a signal.

In conjunction with the sensor system, a specific embodiment of theabove-mentioned device may be employed or utilized advantageously tocontrol and/or execute the monitoring of the surroundings. The devicemay include a plurality of mechanisms, which may be situated in at leastone unit of the sensor system. According to one specific embodiment, theunits of the sensor system include the first detector, the seconddetector, optionally, at least one further detector, and a datamanagement device.

According to one specific embodiment, the first detector and the seconddetector may have an identical or a different detection principle. Sucha specific embodiment provides the advantage that mutualplausibility-checking and, additionally or alternatively,supplementation and concretization of acquired data are renderedpossible.

In particular, the first detector may have a detection principle basedon lidar. In this connection, the second detector may have a detectionprinciple based on radar or may be implemented as a vehicle camera. Sucha specific embodiment provides the advantage that on one hand, rapid andexact scanning of the surroundings may be achieved, which may beavailable for pre-parameterization of an assistance system, and, on theother hand, for checking the plausibility of, and concretizing thisscanning, in order to achieve accurate and reliable monitoring of thesurroundings.

The sensor system may also have a further detector. The further detectormay be connected to the device so as to be able to transmit a signal.The further detector may have a detection principle identical to ordifferent from the first detector and, additionally or alternatively,the second detector. Such a specific embodiment provides the advantagethat a plurality of detectors may be considered and/or used in themonitoring of the surroundings, in order to further increase an accuracyand reliability of the monitoring of the surroundings.

A computer program product or computer program including program code,which may be stored in a machine-readable carrier or storage medium,such as a solid state memory, a hard disk storage device or an opticalstorage device and is used for performing, implementing and/orcontrolling the steps of the method according to one of theabove-described specific embodiments, in particular, when the programproduct or program is executed on a computer or a device, is alsoadvantageous. Therefore, such a storage device is also claimed. Adevice, which is configured to execute one of the above-mentionedmethods, is claimed, as well.

Exemplary embodiments of the approach put forward here are illustratedin the drawings and explained in greater detail in the followingdescription.

In the following description of the exemplary embodiments of the presentinvention, the same or similar reference numerals are used for theelements that are shown in the different figures and function similarly,in which case a repeated description of these elements is omitted.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a schematic representation of a vehicle having a sensorsystem according to an exemplary embodiment.

FIG. 2 shows a schematic representation of a sensor system according toan exemplary embodiment.

FIG. 3 shows a flow chart of a method for monitoring a surrounding areaaccording to an exemplary embodiment.

FIG. 4 shows a schematic representation of a vehicle having a sensorsystem according to an exemplary embodiment.

DETAILED DESCRIPTION

FIG. 1 shows a schematic representation of a vehicle 100 having a sensorsystem 110 according to an exemplary embodiment. According to theexemplary embodiment depicted here, vehicle 100 is a road vehicle, forexample, a passenger car, a cargo truck or another commercial vehicle,in particular, a vehicle for highly automated traveling. By way ofexample, only an assistance system 105 and sensor system 110 of vehicle100 are shown. Assistance system 105 is configured to control or executean assistance function and/or driving function of vehicle 100. Sensorsystem 110 is configured to carry out monitoring of the surroundings forvehicle 100. Vehicle 100 is situated inside of a surrounding area. Anobject OBJ is located in the surroundings of vehicle 100.

In vehicles such as the vehicle 100 shown, optical, but alsoreflection-based systems, such as radar sensors, lidar sensors,ultrasonic sensors and camera systems, should be able to detectmovements of objects and other dynamic effects, such as discontinuousroad markings, events suddenly occurring, such as a load falling from avehicle traveling ahead, but also randomly occurring faults in thesystem, within a defined period of time. In particular, the utilizedmeasuring principles of surround sensors, but also algorithms, whichgenerate signals for electronic data processing from physical and/orelectrical data, should be synchronized to a system time.

Due to different transit times of a measuring medium, there are firsttemporal inaccuracies in real-time data acquisition, which may beprevented in accordance with exemplary embodiments. In the case of lidarand cameras, light is used as a measuring medium. In accordance withmeasuring system technology, the camera measures a quantity of incidentlight, light intensity, etc. A measurement takes place with the aid of aso-called imager of the camera. In the case of lidar, a quantity ofreflected light of an emitted laser pulse is measured. In thisconnection, detection may take place very rapidly with the aid ofphysical-to-electrical conversion, e.g., comparable to the principle ofa photodiode. Using the delay time, different characteristics may thenbe detected in combination with scattering effects. This may take placein a manner similar to Raman lidar systems or differential absorptionlidar. Radar and ultrasonics are based on the measurement of an emittedmeasuring medium and its reflection, as well. In the case of the camera,if only the simple path from the object to the measuring unit isrelevant, then, in the case of lidar, it is twice the distance that thelight covers from the emitter to the object, and back to the receiver.Since this takes place approximately at the speed of light, tolerancesoccurring in the measuring system, and also the physical transit time ofthe light beam in the actual surrounding area, are very low. In the caseof ultrasonics and in the case of radar, the transit times are longer,and, in particular, tolerances in the physical signal propagation time,and a time for physical-to-electrical conversion of the information, aresubjected to tolerances due to environmentally dependent effects.

In measuring systems, measures for signal filtering, error correctionand error control, signal conversion, and algorithms for determiningcharacteristics, are temporally highly variable and dependent onenvironmental parameters, such as temperature, vibrations, stresses,electromagnetic compatibility, etc. In the case ofphysical-to-electrical conversion, a time stamp is normally applied tothe electrical data, in order to render such data delay timestechnically measurable and evaluable.

The sensor system 110 shown includes a first detector 120, a seconddetector 130 and a device 160 for monitoring the surroundings. Accordingto the exemplary embodiment represented here, sensor system 110optionally includes a further or third detector 140, as well. Device 160is connected to detectors 120, 130, 140 so as to be able to transmit asignal. According to the exemplary embodiment represented here, sensorsystem 110 further includes a data management device 150 in the form ofa so-called deadline monitor or the like. In this context, datamanagement device 150 is connected to detectors 120, 130, 140 so as tobe able to transmit a signal.

According to different exemplary embodiments, detectors 120, 130, 140have identical detection principles or at least partially differentdetection principles. According to the exemplary embodiment representedhere, detectors 120, 130, 140 have different detection principles. Inparticular, first detector 120 has a detection principle based on lidar.Second detector 130 has, for example, a detection principle based onradar. Third detector 140 is constructed, for example, as a vehiclecamera or has a detection principle based on radar, as well.

First detector 120 is assigned a first reading-in device 122 and a firstexecution device 124 or first processing device 124. First reading-indevice 122 is configured to read in a first surrounding-area informationitem 121 with the aid of first detector 120. First surrounding-areainformation item 121 represents an information item, which is receivablefrom the surroundings of vehicle 100 and is about object OBJ in thesurroundings of vehicle 100. In addition, first reading-in device 122 isconfigured to pass on read-in, surrounding-area information item 121 tofirst execution device 124 or to prepare it for output. First executiondevice 124 is configured to carry out processing of firstsurrounding-area information item 121 with the aid of first detector120, in order to provide sensor data 125. According to the exemplaryembodiment represented here, first detector 120 is further assigned afirst emitting device 126. First emitting device 126 is configured toemit a first sensing signal 127 into the surroundings of vehicle 100with the aid of first detector 120. In response to first sensing signal127, first surrounding-area information item 121 is receivable from thesurroundings of vehicle 100, that is, from object OBJ. First sensingsignal 127 is, for example, a light signal, in particular, a lasersignal or lidar signal. Thus, first surrounding-area information item121 is a reflected light signal.

Second detector 130 is assigned a second reading-in device 132 and asecond execution device 134 or second processing device 134. Secondreading-in device 132 is configured to read in a second surrounding-areainformation item 131 with the aid of second detector 130. Secondsurrounding-area information item 131 represents an information item,which is receivable from the surroundings of vehicle 100 and is aboutobject OBJ in the surroundings of vehicle 100. In addition, secondreading-in device 132 is configured to pass on read-in, surrounding-areainformation item 131 to second execution device 134 or to prepare it foroutput. Second execution device 134 is configured to carry outprocessing of second surrounding-area information item 131 with the aidof second detector 130, in order to provide second sensor data 135.According to the exemplary embodiment represented here, second detector130 is further assigned a second emitting device 136. Second emittingdevice 136 is configured to emit a second sensing signal 137 into thesurroundings of vehicle 100 with the aid of second detector 130. Inresponse to second sensing signal 137, second surrounding-areainformation item 131 is receivable from the surroundings of vehicle 100,that is, from object OBJ. Second sensing signal 137 is, for example, aradio signal, in particular, a radar signal. Thus, secondsurrounding-area information item 131 is a reflected radar signal.

The third detector 140 provided in accordance with the exemplaryembodiment represented here is assigned a third reading-in device 142and a third execution device 144 or third processing device 144. Thirdreading-in device 142 is configured to read in a third surrounding-areainformation item 141 with the aid of third detector 140. Thirdsurrounding-area information item 141 represents an information item,which is receivable from the surroundings of vehicle 100 and is aboutobject OBJ in the surroundings of vehicle 100. In addition, thirdreading-in device 142 is configured to pass on read-in, surrounding-areainformation item 141 to third execution device 144 or to prepare it foroutput. Third execution device 144 is configured to carry out processingof third surrounding-area information item 141 with the aid of thirddetector 140, in order to provide third sensor data 145. According tothe exemplary embodiment represented here, third detector 140 isassigned a third emitting device 146, if third detector 140 has, forexample, a detection principle based on radar. Third emitting device 146is configured to emit a third sensing signal 147 into the surroundingsof vehicle 100 with the aid of third detector 140. In response to thirdsensing signal 147, third surrounding-area information item 141 isreceivable from the surroundings of vehicle 100, that is, from objectOBJ. Third sensing signal 147 is, for example, a radio signal, inparticular, a radar signal. Thus, third surrounding-area informationitem 141 is a reflected radar signal.

Device 160 includes reading-in devices 122, 132, 142, execution devices124, 134, 144, and a fusion device 152. According to the exemplaryembodiment represented here, device 160 further includes emittingdevices 126, 136, 146. According to the exemplary embodiment representedhere, device 160 additionally includes a determination device 154.According to one exemplary embodiment, reading-in devices 122, 132, 142,execution devices 124, 134, 144, and/or emitting devices 126, 136, 146are manufactured as parts of detectors 120, 130, 140, respectively.According to the exemplary embodiment represented here, fusion device152 and/or determination device 154 is/are constructed as parts of datamanagement device 150.

Fusion device 152 is configured to merge first sensor data 125 andsecond sensor data 135, or first sensor data 125, second sensor data 135and third sensor data 145, using a time-difference information item 153,in order to provide a measuring signal 155. In other words, fusiondevice 152 is configured to carry out sensor data fusion of first sensordata 125 and/or of second sensor data 135 and/or of third sensor data145. Measuring signal 155 represents monitored surroundings of vehicle100 and/or a detected object in the surroundings of vehicle 100. Fusiondevice 152 is configured to pass on measuring signal 155 to assistancesystem 105 and/or to prepare it for output. Time-difference informationitem 153 represents a time difference between a first latency time,which is needed by first detector 120 for the processing, up to theprovision of first sensor data 125, and a second latency time, which isneeded by second detector 130 for the processing, up to the provision ofsecond sensor data 135, and/or a third latency time, which is needed bythird detector 140 for the processing, up to the provision of thirdsensor data 145. According to the exemplary embodiment represented here,determination device 154 is configured to ascertain time-differenceinformation item 153, in particular, to ascertain it continuously. Forthis, determination device 154 is configured to combine signalpropagation times, times for physical-to-electrical conversion, andtimes for signal processing to obtain the first latency time regardingfirst detector 120, to obtain the second latency time regarding seconddetector 130, and to obtain the third latency time regarding thirddetector 140. Furthermore, determination device 154 is configured topass on ascertained time-difference information item 153 to fusiondevice 152 and/or to prepare it for output.

According to one exemplary embodiment, fusion device 152 is configuredto scale first sensor data 125 and/or second sensor data 135 and/orthird sensor data 145, using time-difference information item 153. Inaddition, or alternatively, fusion device 152 is configured tosynchronize first sensor data 125 and/or second sensor data 135 and/orthird sensor data 145, in particular, with respect to each other and/orrelative to a system time of sensor system 110, using time-differenceinformation item 153 and a model for sensor data fusion. Fusion device152 is configured to use a neural network, a classification method, astochastic method, a Kalman filter, fuzzy logic and/or logicaloperations in the sensor data fusion.

According to one exemplary embodiment, using the sensor data 125, 135,145 to be merged, determination device 154 is configured to ascertaintime-difference information item 153 for the sensor data 125, 135, 145to be merged by fusion device 152. According to a further exemplaryembodiment, determination device 154 is configured to ascertaintime-difference information item 153 for the sensor data 125, 135, 145to be emerged by fusion device 152, using temporally previous sensordata 125, 135, 145, for example, using sensor data 125, 135, 145, whichwere provided in a predetermined time window prior to the provision ofthe sensor data 125, 135, 145 to be merged currently by fusion device152. According to a further exemplary embodiment, determination device154 is configured to ascertain time-difference information item 153,using sensor data 125, 135, 145 provided during a calibration phase ofdevice 160.

FIG. 2 shows a schematic representation of a sensor system 110 accordingto an exemplary embodiment. Sensor system 110 corresponds to, or issimilar to, sensor system 110 from FIG. 1. In FIG. 2, a first detector120, a second detector 130 and a data management device 150 of sensorsystem 110 are shown. In FIG. 2, an object OBJ is also put in thesurroundings of a vehicle. In addition, a first surrounding-areainformation item 121, a first sensing signal 127, a secondsurrounding-area information item 131, a second sensing signal 137,first sensor data 125, second sensor data 135, and a measuring signal155 are drawn into FIG. 2.

First detector 120 has lidar as a detection principle. Second detector130 has radar as a detection principle. Data management device 150 isconstructed as a so-called deadline monitor. First measurement data 125are provided with a first latency time t_(Lidar). Second measurementdata 135 are provided with a second latency time t_(Lidar)delta_(Radar)=t_(real). Data management device 150 is configured tocheck a signal integrity, a temporal synchronism and a sequence, as wellas to add data regarding position, trajectory, environmental conditionsand the like. In addition, data management device 150 is configured toprovide a real-time information item with measuring signal 155.

In other words, an example of a deadline monitor as a data managementdevice 150 in the case of diverse object detection and geometricmeasurement by radar and lidar is shown in FIG. 2.

FIG. 3 shows a flow chart of a method 300 for monitoring a surroundingarea according to an exemplary embodiment. Method 300 may beimplemented, in order to monitor the surroundings for a vehicle. In thiscontext, method 300 for monitoring the surroundings in conjunction with,and/or using the sensor system from FIG. 1 or FIG. 2 or a similar sensorsystem, is implementable. In particular, method 300 for monitoring thesurroundings in conjunction with, and/or using the device from FIG. 1 orFIG. 2 or a similar device, is implementable.

In the method 300 for monitoring surroundings, in a reading-in step 310,a first surrounding-area information item is read in with the aid of afirst detector, and a second surrounding-area information item is readin with the aid of a second detector. In this context, the firstsurrounding-area information item and the second surrounding-areainformation item represent an information item, which is receivable fromthe surroundings of the vehicle and is about at least one object in thesurroundings of the vehicle.

Subsequently, in an execution step 320, processing of the firstsurrounding-area information item is carried out with the aid of thefirst detector, in order to provide first sensor data, and processing ofthe second surrounding-area information item is carried out with the aidof the second detector, in order to provide second sensor data.

Subsequently, in turn, in a fusion step 330, the first sensor data andthe second sensor data are merged, using a time-difference informationitem, in order to provide a measuring signal. In this connection, thetime-difference information item represents a time difference between afirst latency time, which is needed by the first detector for theprocessing, up to the provision of the first sensor data, and a secondlatency time, which is needed by the second detector for the processing,up to the provision of the second sensor data. The measuring signalrepresents monitored surroundings of the vehicle. For example, themeasuring signal is provided for output to at least one assistancesystem or other vehicle system of the vehicle.

According to one exemplary embodiment, method 300 for monitoringsurroundings also includes a step of ascertaining the time-differenceinformation item for use in fusion step 330. In this connection, inascertaining step 340, signal propagation times, times forphysical-to-electrical conversion, and times for signal processing arecombined to obtain the first latency time with regard to the firstdetector, and signal propagation times, times for physical-to-electricalconversion, and times for signal processing are combined to obtain thesecond latency time with regard to the second detector. Ascertainingstep 340 may now be executed continuously or repeatedly, in particular,while other steps of method 300 are executed.

According to one exemplary embodiment, method 300 for monitoring thesurroundings includes a step 350 of emitting a first sensing signal withthe aid of the first detector and/or emitting a second sensing signalwith the aid of the second detector, into the surroundings of thevehicle. In response to the first sensing signal, the firstsurrounding-area information item is receivable from the surroundings ofthe vehicle. In response to the second sensing signal, the secondsurrounding-area information item is receivable from the surroundings ofthe vehicle. Emitting step 350 may be executed prior to reading-in step310.

FIG. 4 shows a schematic representation of a vehicle 100 having a sensorsystem according to an exemplary embodiment. The sensor system, which isnot explicitly denoted in FIG. 4, corresponds to, or is similar to thesensor system from FIG. 1 or FIG. 2.

In this connection, the sensor system includes first detector 120,second detector 130, as well as a further first detector 420 and afurther second detector 430. Sensor data from first detector 120 andsecond detector 130 are merged with the aid of fusion device 152 of thesensor system. An algorithm device 456 is connected in outgoing circuitto fusion device 152. Sensor data from further, first detector 420 andfurther, second detector 430 are merged with the aid of a further fusiondevice 452 of the sensor system. A further algorithm device 458 isconnected in outgoing circuit to further fusion device 452.

Algorithm device 456 and further algorithm device 458 are configured toevaluate the respective sensor data. A common channel selection device470 or a so-called gatekeeper 470 is connected in outgoing circuit toalgorithm device 456 and further algorithm device 458. A controllingelement 405 or actuator 405 is, in turn, connected in outgoing circuitto channel selection device 470. Actuator 405 is assigned to a vehiclesystem of vehicle 100 and is also controllable, for example, via anassistance system of vehicle 100.

A safety monitoring device 480 or a so-called safety monitor 480 isconfigured to monitor the sensor data prior to the sensor data fusion byfusion device 152. In addition, safety monitoring device 480 isconfigured to influence a signal transmitted between algorithm device456 and channel selection device 470 as a function of a result ofimplemented safety monitoring. A further safety monitoring device 490 ora further, so-called safety monitor 490 is configured to monitor thesensor data prior to the sensor data fusion by further fusion device152. In addition, further safety monitoring device 490 is configured toinfluence a signal transmitted between further algorithm device 458 andchannel selection device 470 as a function of a result of implementedsafety monitoring.

For example, based on a surrounding-area information item initiallyavailable, it may be assumed that a child is suddenly running in frontof vehicle 100. If it is then discerned by the sensor system, and/or dueto the evaluation of a surrounding-area information item availablelater, that it was not a child, but swirled-up dust or the like, then,for example, a braking command may simply not be put through to actuator405. Then, gatekeeper 470 lets only the correct information, that is,the correct measuring signal, through. This is advantageous for sensorsystems, in particular, redundant systems.

In the following, exemplary embodiments are explained once more in acondensed manner and/or briefly introduced with reference to the figuresdescribed above.

In sensor system 110, one object is to combine physical data delay timesand times for physical-to-electric conversion and to ascertain asystem-dependent time constant in the form of latency times andtime-difference information item 153, as well. If particular factorsresult in significant delay-time variances, then these are acquired, forexample, for the delay time, so that a pair of values from the factorsis available at any time. In addition, the times for signal conversion,error correction and error control are measured. In particular, inreflection-based detectors, that is, systems such as laser and radar,portions of the emitted beams, that is, sensing signals 127, 137, 147,and a time, are measured, which are then converted to the targetvariable, for example, distance and angle, or the like. In the case ofthe various sensors, the determination of the target variable is carriedout in different ways, depending on the measuring principle, and mostlytakes differing amounts of time within the relevant tolerances, whichallows such a chain in the form of a flow of data to vary.

Since the chain made up of a physical effect to be measured, e.g.,movement of an object OBJ, signal propagation time,physical-to-electrical conversion, measuring signal conditioning andprovision of the sensor signal to be transmitted and/or the sensor data125, 135, 140 to be transmitted for a particular detector 120, 130, 140,generally proceeds temporally in a fixed time window, it is provided, inthis case, that a time stamp first be attached to the specific sensorsignal upon the provision of data. In general, a sensor-specific,conservative time window is generated by the different tolerances, whichmay add up and also subtract. As an alternative, a time information itemfrom the infrastructure may be used. For example, a camera may detect achange from red to green; the change being able to be transmittedwirelessly to vehicle 100 as a time reference. Since a traffic light isgeometrically stationary, this may be used advantageously forcalibrating sensor system 110.

In general, with the aid of the execution devices 124, 134, 144, errortreatments, diagnostics and error correction mechanisms are stillnecessary for signal conditioning, in order that sensor data 125, 135,140 in the form of a sufficiently valid signal may be provided forcommunication. In general, this time also lies in a time intervaltypical of the sensor principle. Thus, for the communication, a typicalage of the measurement information may be provided, which is in directrelation to the actual physical effect. In general, another so-calledmessage counter is added, in order that a signal order and lostmessages, as well, may be identified. If detector 120, 130, 140 is alsosynchronized temporally after an initialization, in order that thesystem time and message sequences are synchronized, then each validsensor signal, including the time, may be related to the real, physicaleffect to be measured.

Redundant measuring elements are often used as detectors 120, 130, 140,such as, e.g., in the case of a stereo camera, or even redundantsensors; or in the case of reflection-based systems, echoes are used toobtain redundant and/or plausible data. In this case, the issue arises,that the signals also vary in tolerance, even when the same measuringprinciple is often used, until they are relayed via the communication.Using the information regarding the general time sequence up to thesignal provision, a time window may be generated, which waits for thefirst measured value. Based on the first measured value, an expectedposition may be defined at the second redundant, measured value. If thismeets the expectation, the signal or sensor data 125, 135, 140 may betransferred as established information to the communication, includingthe time stamp. Diagnostics, signal quality or the physical condition ofthe signal may also be assigned to the time stamp. In this connection,in execution step 320, that is, with the aid of execution devices 124,134, 144, sensor data 125, 135, 140 are provided with respective timeinformation items, and in fusion step 330, that is, with the aid offusion device 152, they are merged, using the time information items.

Since a time stamp is generally quite long, sensor data 125, 135, 140may also be synchronized to a message counter or ring counter having amore simple data format. Thus, the message counter constitutes the timeequivalent. Since, in general, a time window for data transmission mayoften be larger than a frequency of incoming sensor data, redundancycomparators may average a variance in the signal propagation timelogically, and consequently, tolerance deviations per redundant signalare reduced. By compensating for signal propagation times typical of asensor, instances of sensor data fusion based on diversely redundantsensors in accordance with exemplary embodiments may be synchronizedtemporally accurately to the actual physical effect. Since a travelingvehicle also moves itself (driving, steering, pitching, vibrations),many factors have an influence on sensor data 125, 135, 140 and therelationship to reality. If a so-called deadline monitor, which ispreconfigured to all relevant data having a particular expectation inthe defined time interval, is used for acquiring the real informationand the relevant environmental and surrounding-area information items,synchronized deterministic data may be generated.

For data security, but also for reasons of data integrity, individualinformation items may be encrypted or marked or the like. Therefore,incorrect or corrupted data may be rejected by the deadline monitor or,in the case of highly available systems, may be denoteddata-technology-specifically by signal qualifiers as having lessintegrity.

This then makes sense, in particular, if unsharp logic, such asso-called fuzzy logic or neural networks, which have different algorithmrunning times, are used, or if a plurality of sources may be used forchecking plausibility. Different delay times, which, in a communicationof a correctness information item, may be properly compensated for andcommunicated more rapidly than an actual useful information item, arealso present in the case of neural networks. Consequently, a signalavailability may be increased. Thus, in the step 320 of carrying out theprocessing, that is, with the aid of execution devices 124, 134, 144,sensor data 125, 135, 140 are provided with an integrity informationitem, which is checked in fusion step 330, that is, with the aid offusion device 152.

The surrounding-area information item and/or the first information iteminitially available or available from a first source may be used, inorder to prepare for degradation in sensor system 110 and then toconduct an active or more intensive vehicle intervention in response toan information item confirmed by a surrounding-area information itemavailable later or available from a second source. For example, ablack-and-white image of a lidar sensor may result in a reduction of thespeed of vehicle 100; a surrounding-area information item from a radarsensor being able to result in deceleration, if the surrounding-areainformation item of a lidar sensor is confirmed. Alternatively, anobject OBJ or obstacle may be identified as not worthy of braking for,and vehicle 100 may resume the original speed.

If an exemplary embodiment includes an “and/or” conjunction between afirst feature and a second feature, then this is to be read such that,according to one specific embodiment, the exemplary embodiment includesboth the first feature and the second feature, and according to anotherspecific embodiment, the exemplary embodiment includes either only thefirst feature or only the second feature.

What is claimed is:
 1. A method for monitoring surroundings of avehicle, the method comprising: reading in a first surrounding-areainformation item with a first detector and a second surrounding-areainformation item with a second detector, wherein the firstsurrounding-area information item and the second surrounding-areainformation item represent an information item, which is receivable fromthe surroundings of the vehicle and is about at least one object in thesurroundings; processing the first surrounding-area information itemwith the first detector, to provide first sensor data, and processing ofthe second surrounding-area information item with the second detector,to provide second sensor data; and merging the first sensor data and thesecond sensor data, using a time-difference information item, to providea measuring signal; the time-difference information item representing atime difference between a first latency time, which is used by the firstdetector for the processing, up to the provision of the first sensordata, and a second latency time, which is used by the second detectorfor the processing, up to the provision of the second sensor data,wherein the measuring signal represent monitored surroundings of thevehicle.
 2. The method of claim 1, further comprising: ascertaining thetime difference information item, wherein in the ascertaining, signalpropagation times, times for physical-to-electrical conversion, andtimes for signal processing are combined to obtain the first latencytime with regard to the first detector and combined to obtain the secondlatency time with regard to the second detector.
 3. The method of claim1, wherein in the fusion, (i) at least one of the first sensor data andthe second sensor data are scaled, using the time-difference informationitem, and/or (ii) the first sensor data and the second sensor data aresynchronized, using the time-difference information item and a model forsensor data fusion.
 4. The method of claim 1, wherein in the fusion, atleast one of a neural network, a classification method, a stochasticmethod, a Kalman filter, fuzzy logic and/or logic operations are used.5. The method of claim 1, wherein in the processing, the first sensordata are provided with a first time information item, and the secondsensor data are provided with a second time information item, andwherein in the fusion, the first sensor data and the second sensor dataare merged, using the first time information item and the second timeinformation item.
 6. The method of claim 1, further comprising: emittinga first sensing signal with the first detector and/or emitting a secondsensing signal with the second detector, into the surroundings of thevehicle, wherein the first surrounding-area information item and/or thesecond surrounding-area information item are receivable from thesurrounding area of the vehicle in response to the first sensing signaland/or in response to the second sensing signal.
 7. The method of claim1, wherein in the processing, the first sensor data and the secondsensor data are provided with an integrity information item, and whereinthe integrity information item is checked in the fusion task.
 8. Themethod of claim 1, wherein in the reading-in, a further surrounding-areainformation item is read in with a further detector, wherein the furthersurrounding-area information item represents an information item, whichis receivable from the surroundings of the vehicle and is about at leastone object in the surroundings, and wherein the processing of thefurther surrounding-area information item is carried out with thefurther detector, to provide further sensor data, and wherein in thefusion, the further sensor data are merged with the first sensor dataand/or with the second sensor data, using the time-differenceinformation item, and wherein the time-difference information itemrepresents a time difference between the first latency time, the secondlatency time and/or a further latency time, which is needed by thefurther detector for the processing, up to the provision of the furthersensor data.
 9. The method of claim 1, further comprising: checking aconsistency of the first surrounding-area information item and thesecond surrounding-area information item; passing on the measuringsignal or the first surrounding-area information item, if there isconsistency between the first surrounding-area information item and thesecond surrounding-area information item; and blocking the measuringsignal or the first surrounding-area information item, if there is noconsistency between the first surrounding-area information item and thesecond surrounding-area information item.
 10. An apparatus formonitoring surroundings of a vehicle, comprising: a device configured toperform the following: reading in a first surrounding-area informationitem with a first detector and a second surrounding-area informationitem with a second detector, wherein the first surrounding-areainformation item and the second surrounding-area information itemrepresent an information item, which is receivable from the surroundingsof the vehicle and is about at least one object in the surroundings;processing the first surrounding-area information item with the firstdetector, to provide first sensor data, and processing of the secondsurrounding-area information item with the second detector, to providesecond sensor data; and merging the first sensor data and the secondsensor data, using a time-difference information item, to provide ameasuring signal; the time-difference information item representing atime difference between a first latency time, which is used by the firstdetector for the processing, up to the provision of the first sensordata, and a second latency time, which is used by the second detectorfor the processing, up to the provision of the second sensor data,wherein the measuring signal represent monitored surroundings of thevehicle.
 11. A sensor system for monitoring surroundings of a vehicle,comprising: a first detector; a second detector; and a device configuredto perform the following: reading in a first surrounding-areainformation item with a first detector and a second surrounding-areainformation item with a second detector, wherein the firstsurrounding-area information item and the second surrounding-areainformation item represent an information item, which is receivable fromthe surroundings of the vehicle and is about at least one object in thesurroundings; processing the first surrounding-area information itemwith the first detector, to provide first sensor data, and processing ofthe second surrounding-area information item with the second detector,to provide second sensor data; and merging the first sensor data and thesecond sensor data, using a time-difference information item, to providea measuring signal; the time-difference information item representing atime difference between a first latency time, which is used by the firstdetector for the processing, up to the provision of the first sensordata, and a second latency time, which is used by the second detectorfor the processing, up to the provision of the second sensor data,wherein the measuring signal represent monitored surroundings of thevehicle.
 12. The sensor system of claim 11, wherein the first detectorand the second detector have an identical or a different detectionprinciple.
 13. The sensor system of claim 11, wherein the first detectorhas a detection principle based on lidar; and the second detector has adetection principle based on radar or is constructed as a vehiclecamera.
 14. The sensor system of claim 11, further comprising: a furtherdetector connected to the device so as to be able to transmit a signal.15. A non-transitory computer readable medium having a computer program,which is executable by a processor, comprising: a program codearrangement having program code for monitoring surroundings of avehicle, by performing the following: reading in a firstsurrounding-area information item with a first detector and a secondsurrounding-area information item with a second detector, wherein thefirst surrounding-area information item and the second surrounding-areainformation item represent an information item, which is receivable fromthe surroundings of the vehicle and is about at least one object in thesurroundings; processing the first surrounding-area information itemwith the first detector, to provide first sensor data, and processing ofthe second surrounding-area information item with the second detector,to provide second sensor data; and merging the first sensor data and thesecond sensor data, using a time-difference information item, to providea measuring signal; the time-difference information item representing atime difference between a first latency time, which is used by the firstdetector for the processing, up to the provision of the first sensordata, and a second latency time, which is used by the second detectorfor the processing, up to the provision of the second sensor data,wherein the measuring signal represent monitored surroundings of thevehicle.